<html>
<head><meta charset="utf-8"><title>dynamic linkage with cdylibs · t-compiler · Zulip Chat Archive</title></head>
<h2>Stream: <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/index.html">t-compiler</a></h2>
<h3>Topic: <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/dynamic.20linkage.20with.20cdylibs.html">dynamic linkage with cdylibs</a></h3>

<hr>

<base href="https://rust-lang.zulipchat.com">

<head><link href="https://rust-lang.github.io/zulip_archive/style.css" rel="stylesheet"></head>

<a name="181581986"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/dynamic%20linkage%20with%20cdylibs/near/181581986" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Matthew Maurer <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/dynamic.20linkage.20with.20cdylibs.html#181581986">(Nov 21 2019 at 20:41)</a>:</h4>
<p><a href="https://doc.rust-lang.org/reference/linkage.html" target="_blank" title="https://doc.rust-lang.org/reference/linkage.html">https://doc.rust-lang.org/reference/linkage.html</a> <a href="https://github.com/rust-lang/rust/issues/4" target="_blank" title="https://github.com/rust-lang/rust/issues/4">#4</a> seems to imply that dynamically linked rust crates would be available in the cdylib case, though it just says dynamic library, so perhaps it refers only to rust dylibs.<br>
<a href="https://github.com/rust-lang/rust/blob/master/src/librustc_metadata/dependency_format.rs#L79" target="_blank" title="https://github.com/rust-lang/rust/blob/master/src/librustc_metadata/dependency_format.rs#L79">https://github.com/rust-lang/rust/blob/master/src/librustc_metadata/dependency_format.rs#L79</a> indicates otherwise, stating that both staticlibs and cdylibs must have all static rust crates.</p>
<p>In my group's use case, it is important that memory usage be kept to a minimum. One way we intended to tackle this was for the standard library (and other components, but the standard library is an obvious large piece) to have their code size costs amortized across all the places it is linked in by using dynamic linking. Unfortunately, we also intended to use cdylibs to gradually introduce Rust into a mostly C++ codebase, and these two concerns are now at odds.</p>
<p>Is there a rationale for this decision, or is support simply unimplemented? The potential issues I can see are:</p>
<ul>
<li>Two cdylibs could end up accessing the same cratewide globals should those exist. I don't see a mitigation for this, but I'd also hope most crates don't have such complicated contracts on their interfaces that this would become an issue.</li>
<li>If the cdylibs were built by Cargo, they could have differently configured dylibs resulting in conflict. In our case, a monolithic build system is in use rather than cargo, so two cdylibs asking for a particular dependency will be given an identical artifact.</li>
<li>If the two cdylibs used different allocators, this could generate a conflict, but again, we intend to build all cdylibs with the same allocator.</li>
</ul>
<p>If these are non-issues, we could consider allowing dynamic linking in general for staticlibs and cdylibs, but still defaulting to static linking for ease of use.<br>
If some of these are potential issues, we could possibly mask this off at the Cargo layer (because separately combining cdylibs with dynamic linkage from two cargo invocations could lead to some of the issues above), but allow it at the rustc layer so that monolithic build systems could use this functionality.</p>
<p>I'd also love to hear about any other gotchas you think we might run into if we tried to do dynamic linkage of rust crates into a cdylib.</p>



<a name="181584242"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/dynamic%20linkage%20with%20cdylibs/near/181584242" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> pnkfelix <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/dynamic.20linkage.20with.20cdylibs.html#181584242">(Nov 21 2019 at 21:06)</a>:</h4>
<p>I am so close to diving in and trying to force myself to explore all the points in this design space. (And how would I force myself to do so? In my usual fashion: By composing a blog post that takes me a month to write...!)</p>



<a name="186053581"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/dynamic%20linkage%20with%20cdylibs/near/186053581" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Matthew Maurer <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/dynamic.20linkage.20with.20cdylibs.html#186053581">(Jan 19 2020 at 20:11)</a>:</h4>
<p>If I wanted to try to work on this, would the correct first step be to file an issue or a RFC?</p>



<a name="186055702"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/dynamic%20linkage%20with%20cdylibs/near/186055702" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Matthew Maurer <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/dynamic.20linkage.20with.20cdylibs.html#186055702">(Jan 19 2020 at 21:02)</a>:</h4>
<p>cc <span class="user-mention" data-user-id="116015">@Alex Crichton</span> since he added the test checking that cdylib deps are forced static when he added cdylib support</p>



<a name="186216123"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/dynamic%20linkage%20with%20cdylibs/near/186216123" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/dynamic.20linkage.20with.20cdylibs.html#186216123">(Jan 21 2020 at 19:11)</a>:</h4>
<p><span class="user-mention" data-user-id="132491">@Matthew Maurer</span> as you can imagine there's a lot of tricky issues with dynamic libraries, and what we have now is basically "this works" rather than "this was meticulously designed and is opinionated as to what we should do"</p>



<a name="186216144"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/dynamic%20linkage%20with%20cdylibs/near/186216144" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/dynamic.20linkage.20with.20cdylibs.html#186216144">(Jan 21 2020 at 19:11)</a>:</h4>
<p>in that sense I think tweaking things is fine, but it's notoriuosly hard to do so because there's so many subtelties involved here</p>



<a name="186216215"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/dynamic%20linkage%20with%20cdylibs/near/186216215" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/dynamic.20linkage.20with.20cdylibs.html#186216215">(Jan 21 2020 at 19:12)</a>:</h4>
<p>a lot of the subtelties aren't obvious and makes some decisions seem wonky, so I can try help clarify anything existing if it seems weird</p>



<a name="186223881"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/dynamic%20linkage%20with%20cdylibs/near/186223881" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Matthew Maurer <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/dynamic.20linkage.20with.20cdylibs.html#186223881">(Jan 21 2020 at 20:30)</a>:</h4>
<p><span class="user-mention" data-user-id="116015">@Alex Crichton</span> <br>
tl;dr<br>
Can you clarify why we cannot link rust dylibs into a cdylib? If not, should I just upload a PR to remove this restriction, or is there a process I should go through to test or gain consensus?</p>
<p>(non-tl;dr below)</p>
<p>The specific decision that seems wonky is that if I build a cdylib and try to -C prefer-dynamic or even use --extern to point it at a crate which only has a dylib form, current rustc will reject this linkage pattern. I tried simply removing this restriction, and it Seems To Work (TM) (Patch is at home, I'll upload it after work for reference purposes.)</p>
<p>The justification for wanting this is to allow amortized disk and RAM cost of large rust components like the standard library in an environment where Rust and C++ are coexisting.</p>
<p>I enumerated the things I thought might go wrong above in my intro post for the thread, but:</p>
<ul>
<li>None of these would become an issue unless a user specifically tried to make it happen.</li>
<li>We could consider masking this feature off at the cargo layer to make it even safer for normal users, but still allow direct rustc usage.</li>
</ul>



<a name="186233824"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/dynamic%20linkage%20with%20cdylibs/near/186233824" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/dynamic.20linkage.20with.20cdylibs.html#186233824">(Jan 21 2020 at 22:12)</a>:</h4>
<p><span class="user-mention" data-user-id="132491">@Matthew Maurer</span> I believe we can just remove that as you've seen</p>



<a name="186233832"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/dynamic%20linkage%20with%20cdylibs/near/186233832" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/dynamic.20linkage.20with.20cdylibs.html#186233832">(Jan 21 2020 at 22:12)</a>:</h4>
<p>I always that that was the case!</p>



<a name="186233846"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/dynamic%20linkage%20with%20cdylibs/near/186233846" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Alex Crichton <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/dynamic.20linkage.20with.20cdylibs.html#186233846">(Jan 21 2020 at 22:12)</a>:</h4>
<p>I didn't realize we had a hard check rejecting dylibs linked into cdylibs</p>



<a name="186236960"></a>
<h4><a href="https://rust-lang.zulipchat.com#narrow/stream/131828-t-compiler/topic/dynamic%20linkage%20with%20cdylibs/near/186236960" class="zl"><img src="https://rust-lang.github.io/zulip_archive/assets/img/zulip.svg" alt="view this post on Zulip" style="width:20px;height:20px;"></a> Matthew Maurer <a href="https://rust-lang.github.io/zulip_archive/stream/131828-t-compiler/topic/dynamic.20linkage.20with.20cdylibs.html#186236960">(Jan 21 2020 at 22:53)</a>:</h4>
<p>OK, thanks. I thought there might be something subtle and wanted to check first. I'll send a PR when I get home.</p>



<hr><p>Last updated: Aug 07 2021 at 22:04 UTC</p>
</html>